fix: harden middlecache downloads and consume OpenAPI archive - #33085
Open
mvvmm wants to merge 8 commits into
Open
fix: harden middlecache downloads and consume OpenAPI archive#33085mvvmm wants to merge 8 commits into
mvvmm wants to merge 8 commits into
Conversation
The 24MB openapi.json was fetched over HTTP during the build. With default Accept-Encoding, middlecache served on-the-fly brotli, which has no integrity checksum; a corrupted/truncated transfer silently decompressed to garbage, intermittently failing the build with "Bad control character in string literal in JSON". - downloadToDotTempIfNotPresent now requests identity encoding (no brotli), writes atomically via a temp file, checks response.ok and Content-Length, and retries up to 3 times. An optional validate callback makes the whole download+validate unit retryable. - getSchema now prefers the gzip-compressed openapi.tar.gz from middlecache (extracted with tar, validated by gzip CRC32 + JSON.parse before use), falling back to the raw openapi.json. - Adds unit tests covering retry, validation, size mismatch, and existing-file handling.
Contributor
|
This pull request requires reviews from CODEOWNERS as it changes files that match the following patterns:
|
🚀 Deploying Preview to Cloudflare 🚀Preview URL: https://consume-openapi-archive.previews.developers.cloudflare.com (commit fa057e7)This URL reflects your latest Preview deploymentPreview Deployments by commit
|
bin/fetch-skills.ts had its own inline tar spawn; extractTarGz (shared with the OpenAPI archive path) now supports stripComponents and fetch-skills uses it instead. Adds extractTarGz unit tests.
Contributor
|
Preview URL: https://fa057e71.preview.developers.cloudflare.com |
The build failed intermittently with ENOENT on openapi.json because getSchema downloaded the schema lazily during prerender. Pages render in parallel, so concurrent downloads raced on the shared .tmp file — a retry's cleanup could delete a sibling call's freshly-written file. Mirror the skills flow: a new bin/fetch-openapi.ts (run from the prebuild/predev hooks) downloads the gzip-compressed openapi.tar.gz from middlecache and extracts openapi.json to .tmp before the build starts. getSchema now just reads the local file. - bin/fetch-openapi.ts: download archive (fall back to raw openapi.json), extract + parse as the integrity check; --soft for predev - package.json: prebuild/predev run fetch-openapi after fetch-skills - api.ts: getSchema reads .tmp/.../openapi.json, single-flighted - custom-loaders.ts: getDotTmpPath now resolves the repo root by walking up to package.json (tsx prebuild and the bundled prerender resolve import.meta.url to different depths); downloadToDotTempIfNotPresent dedups concurrent same-destination downloads - tests: concurrent-download dedup + retry-after-failure coverage Verified: full `pnpm run build` succeeds (8932 pages), check/lint/tests pass.
Match bin/fetch-skills.ts behavior: print a skip message and exit when the extracted openapi.json is already present, with --force to re-fetch.
Workers Builds invokes `pnpm run build:incremental`, which skipped the `prebuild` hook, so `bin/fetch-openapi.ts` never ran and prerendering failed with ENOENT on the schema file. Add a `prebuild:incremental` hook mirroring `prebuild` (fetch-skills + fetch-openapi) and make `getSchema` read-only so a build invoked without the pre-step fails loudly instead of silently downloading mid-render. Also factor the middlecache fetch/extract logic into `src/util/openapi-schema.ts` shared by `bin/fetch-openapi.ts` and `getSchema`.
mvvmm
marked this pull request as ready for review
August 28, 2026 17:41
Contributor
Review✅ No issues found in commit Code ReviewThis code review is in beta and may not always be helpful — use your judgment. No code review issues found. ConventionsNo convention issues found. Style Guide ReviewNo style-guide issues found. CommandsOnly codeowners can run commands. Post a comment with the command to trigger it.
|
- custom-loaders: reject tar members with `..` segments or absolute paths (zip-slip); reject on spawn `error` instead of hanging; capture tar stderr in extraction errors - openapi-schema: extract to a staging dir and atomically promote on success, so a failed extract/parse leaves no stale files to mask a fresh download failure - tests: check spawnSync status when building fixtures; add regression tests for unsafe tar member paths - package.json: share the fetch-skills + fetch-openapi command via a single fetch:assets script used by prebuild and prebuild:incremental
Replaces the `response.body!` non-null assertion in downloadWithRetry with an explicit check, so a bodyless response (e.g. 204/HEAD) fails with a clear error instead of an unhelpful TypeError from Readable.fromWeb(null). Adds a regression test.
kodster28
approved these changes
Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes intermittent production build failures around the 24MB
openapi.jsonfetched from middlecache during the Astro prerender. Two root causes:Accept-Encoding, middlecache served on-the-fly brotli (no integrity checksum); a corrupted/truncated transfer silently decompressed to garbage, failing the build withBad control character in string literal in JSON.getSchemadownloaded lazily during prerender; pages render in parallel, so concurrent downloads raced on the shared.tmpfile and a retry's cleanup could delete a sibling call's freshly-written file (ENOENTonopenapi.json).Changes:
bin/fetch-openapi.ts(new, mirrors the skills flow) — runs from theprebuild/predevhooks, downloads the gzip-compressedopenapi.tar.gzfrom middlecache (extracting + parsing as the integrity check), falling back to rawopenapi.json.--softfor predev.getSchemanow just reads the local.tmp/.../openapi.jsonduring prerender — no network, no race.downloadToDotTempIfNotPresentrequestsAccept-Encoding: identity(no brotli), writes atomically (temp file + rename), checksresponse.ok+Content-Length, retries up to 3×, dedups concurrent same-destination downloads, and accepts an optionalvalidatecallback.getDotTmpPathresolves the repo root by walking up topackage.json, so tsx prebuild and the bundled prerender (which resolveimport.meta.urlto different depths) agree on.tmp.extractTarGzhelper (withstripComponents) and unifiesbin/fetch-skills.tsonto it.A companion change in the middlecache pipeline publishes the
openapi.tar.gzarchive (already merged); the prebuild falls back to rawopenapi.jsonif it is absent.Verified: full
pnpm run buildsucceeds (8932 pages),pnpm run check, ESLint, Prettier, and 136/136 tests pass.